U.S. Pat. No. 5,974,348xe2x80x94System and method for performing mobile robotic work operations
The current patent references this as prior art. This patent describes a superficially similar device, but does not contain the three basic elements to a successful flexible automation system, namely: (a) sensing, (b) utilization of sensors by computers or some types of electronics, to (c) adaptively guide robot arms and machinery. Additionally, this prior art makes use of on the ground beacons, and does not provide for a human operator as integral to the operation either locally or remotely.
U.S. Pat. No. 6,052,647xe2x80x94Method and system for automatic control of vehicles based on carrier phase differential GPS.
The current patent may incorporate this patent for controlling the attitude of the robot on uneven terrain.
U.S. Pat. No. 6,199,000xe2x80x94Methods and apparatus for precision agriculture operations utilizing real time kinematic global positioning system systems.
The current patent improves on this prior art by using a combination of sensing and computing to refine the position.
U.S. Pat. No. 6,374,538xe2x80x94Vineyard apparatus, system, and method for vineyard mechanization
The current patent improves on this prior art by using robotics, sensing and computing or artificial intelligence techniques and other electronics to provide flexibility and selectivity on a plant by plant basis. The named patent does this by a large number of well designed mechanical attachments.
U.S. Pat. No. 6,415,229xe2x80x94System for position determination of mobile objects, in particular vehicles.
The current patent improves on this prior art by using sensing and computing for guidance in addition to this system.
Not Applicable
Not Applicable
1. Field of Invention
This invention relates to the application of integrated flexible robotics techniques from industry to agriculture, specifically to the use of robotic armatures, a computer or artificial intelligence system that can sense and decide before acting on the work object, alerting a human operator where intervention is required coupled with, machine vision, laser rastering, radar, infrared, ultrasound, touch or chemical sensing.
2. Prior Art
The genesis of this invention came with the observation of a problem in agriculture with labor costs. This was confirmed by various scholars such as these who stated: xe2x80x9cBecause of the increasingly high cost and scarcity of experienced pruners, efforts have been underway since the late 1960s to mechanize grapevine pruning in the United States. To date, these efforts have proved unacceptable because commercial mechanized pruning devices lack selectivity.xe2x80x9d [University of Michigan Extension, Pruning Grapevines in Michigan, Gordon S. Howell and R. Keith Striegler MSU Department of Horticulture]
This quote summarizes for the viticulture agricultural niche the problems with purely mechanical methods lacking an intelligent robotic component that can sense and decide. Many areas of agriculture today still require large amounts of fairly skilled labor to plant, cultivate, prune, harvest and spray crops. The problem is how to accomplish that? This flexible agricultural robot provides a solution to this class of problem.
U.S. Pat. No. 6,374,538 contains an excellent discussion of the problems with prior art in the area of refined agricultural machinery as it relates to viticulture. This discussion summarizes the drawbacks of a large number of patents granted all of which are purely mechanical methods. This patent, is, to date, the most complete work for purely mechanical methods of viticulture pruning.
All purely mechanical approaches to problems in agriculture such as vineyard or orchard pruning and harvesting, ground crop harvesting or weeding suffer from the inability of mechanical methods to easily adapt to variant conditions without human operator intervention. Vineyards are able to harvest by mechanical methods, and many do. However, mechanical methods are not practical for harvesting of grapes in intact clusters. To date, pruning, harvesting of intact clusters, suckering and tying of vineyards is practiced manually virtually without exception. This is extremely costly for vineyards since it is very labor intensive. The labor-intensive nature of this type of farming also puts the farmers at risk from union activity. Similar problems pertain to fruit orchards, which share a similar business model.
In other situations, for instance in organic farming, weeds must be removed without use of herbicides. This again, results in the farmer making use of a large amount of expensive labor. Organic farming also suffers problems from insect pests that can, in some cases be controlled during critical parts of their life cycle by removal of leaves, and affected material. These types of tasks can never be accomplished by purely mechanical means because purely mechanical methods are not able to intelligently respond to their environment on a selective beyond a very simple level of complexity. In addition, farming in general is receiving greater and greater pressure to minimize use of chemical methods because of environmental issues.
There have been devices, such as that of Slaughter and Lamm at UC Davis in 2000 for machine vision based weed control, which have not been patented. (Found by a thesis search.) However, their system did not control a robot arm, but simply controlled a sprayer, which required precise timing of the motion of the device as it was dragged behind a tractor. Nor did this system perform any but the most rudimentary machine vision operations. They did not attempt to make use of more than one modality for sensing, nor did they consider any of the other elements that make the present invention unique such as intelligent integration of the operator or semi-autonomous limited self guidance during its primary duty execution.
There are other devices, which have been known for a considerable period in industrial automation, which couple robot arms with machine vision for performing tasks, even with a high degree of selectivity, and flexible robot systems which are trainable and easily programmable. However, these are not mobile machines, they are unable to rove around or be steered by an operator. Nor do they incorporate a communications system which allows the experience of each machine to be collated so as to improve the performance of all. Nor, as far as can be determined, have any of these environmentally responsive robotic techniques have been applied to agriculture to date. There are other innovations, such as the use of multiple sensor modalities (i.e. combining machine vision with radar data to produce a better 3 dimensional mapping of an object) which are not typically used in industry, but which can be important in field applications to produce a robust system. Another innovation is the use of GPS incorporated into the robot arms themselves as a way of tracking the motion of the robot""s arms.
This is a system integration invention, which brings together in a new way components which are available and used today in various settings, mostly industrial facilities of various kinds. In addition to improving the labor situation, it results in new capacities which are not available now, such as detailed mapping of croplands traversed. Accordingly, this invention is novel and not obvious within the field of agriculture.
Some aspects of this invention, such as the use of GPS to track motion of robot arms are novel within the field of robotics. Another novel aspect of this invention within the field of robotics is the use of a communications system connected to the manufacturer or manufacturer""s agent which can be used to collect sensing, decision and error information so that other robots can make use of the solutions created for one to benefit all. Current systems for industry, where they have such communications, use them for maintenance and diagnostics purposes. A probable reason why this aspect of the invention would occur for agriculture first is that robots sold for industrial purposes are sold to widely varying competitors, who use them to execute proprietary designs and trade secrets, consequently, such a feature would not be accepted. Agriculture is structured differently.
It is a highly integrated commodity business with a very high degree of cooperation between producers. Accordingly, there are aspects of this invention which are novel and not obvious within the general field of robotics.
The overall object of the invention is to provide a means for replacing human labor on farms in situations where mechanical designs, which cannot make an intelligent decision, will not work. The invention is a flexible robot platform which can be reconfigured by software and changes to attachments. The overall advantages are primarilyxe2x80x94the work can be done with a higher degree of precision and repeatability (which translates to higher quality) at a fraction of the current cost. The machine is also flexible and reconfigurable which makes it more widely useful, as CNC machines are in industry. And, that the farmer can have a higher level of information available about his farm than he otherwise would have, which can be reported to him by the machine.
(a) To provide a system that can sense the precise location of crop plants, their branches, stems, and trunks and if necessary their internal structure through the application of machine vision, ultra wide band radar, ultrasound imaging, infrared imaging, laser rastered 3-D surface mapping, chemical and touch sensors. The integration of data from one or multiple channels, and the possible iterative refinement of sensing using multiple means of sensing for areas identified as critical by the robot is a core element of what makes this invention useful and practical for farmers.
(b) To integrate the multi-channel data into a three dimensional map of each plant. This data is then available for the application of algorithms that can determine where a plant should be cut, sprayed, tied, or otherwise manipulated. This provision creates a means for generalized representation.
(c) To provide a subsystem which will determine, based on locations identified in the three-dimensional map of each plant, how to move a robot arm to that location as rapidly as possible. The advantage of this subsystem is to make it eliminate the task of programming each step of the robotic arm movement as is usually done, replacing it with a simple target location.
(d) To provide a subsystem that will perform a specific operation with the arm after it has been moved to the location specified. This operation can be a highly variable scripted operation. The essence of this object to the invention is its configurable application to robotics in agriculture. Its advantage is that it decouples the task of specifying what to do from the task of getting to the location where the task needs to be done.
(e) To provide a utility subsystem consisting of computers, artificial intelligence, or fizzy logic electronics in some combination that can be used to recognize rapidly situations in which the robot finds itself. This system will fail over to an operator display that will allow the operator to look at what the robot is having a problem with so that the operator can provide a solution to this situation. The utility subsystem may record or learn from this solution and be able to use it again the next time that a similar situation is encountered. The advantage of this is that the robot will gain flexibility as it is used and encounters novel situations outside of the bounds thought up by the original system creators. This feature is important to creating a successful roving robot designed to deal with the variability of plants.
(f) To provide a system that will, both automatically and with operator intervention, record the location of each crop plant, along with diagnostic information about the plant. This information is entered into a database that the farmer can browse to take special action in his croplands.
(g) To provide a system that will alert the operator when a situation is encountered that the machine is not entirely sure how to deal with, presenting the information it has available in an optimized fashion for the operator to figure out and decide. These decisions of the operator will be recorded and processed by a learning algorithm.
(h) To provide a system which will allow the operator to see a representation of the field being worked on, and flag, or make notes for future use, about a particular plant or location in the field.
(i) To provide a subsystem which will communicate with a central site data repository via radio, cellular phone, satellite or some other similar technology. Use this link to report:
a. on the crop data. It also allows the farmer to have his cropland information recorded in a standard shareable form. This can be an important advantage to farms in cooperatives and multi-farm operations comprising tens of thousands of acres across multiple states.
b. any learning overrides that the robot has received from its operator. The advantage of this is that it enables each robot to be used as a learning and refinement station for all other robots in use for that crop. This is a very important advantage for a flexible field robot to have if it is to be successful.
c. on request of the central site, journal records on the detail operation of the computer and Al systems on the robot. The advantage of this is that it allows for iterative refinement of the system using live data from the field. A major problem with such situations is the difficulty involved in getting the data necessary to diagnose and fix problems that arise.
d. movement and use of the robot. This allows the robot to be tracked for liability purposes, and makes it hard to steal the equipment as well. This provides advantages to both manufacturer and farmer for insurance purposes.
e. self diagnostic information from the robot. The advantage of this is that it allows replacement parts and warranty service to be scheduled based on real information rather than just on a schedule since reliability is so important for farmers.
(j) To provide a self powered wheeled or tracked platform vehicle that can be steered by an operator across varying terrain, with the machine providing leveling and fine guidance so that virtually no steering at all is required while operating along a crop row. This simplifies the task of guiding the vehicle greatly, and will prevent the fairly common occurrence of collision of the vehicle with crop plants when an operator makes a mistake. It also makes it possible for the vehicle to be used on a variety of terrain and maintain itself horizontal to the growth of the plants, thereby providing a more useful base for attachment of the robotic arms. This also frees up the operator for the important task of aiding the robot in performing its intended duties should a situation arise which the robot is unable to solve.
(k) To provide a self powered platform that is able to record and repeat a track taken through a field, so that the operator is only required to steer it to the starting point in the field, after which the self powered platform takes over. This simplifies the task of operation and allows the operator to pay attention to exception conditions.
(l) To provide GPS incorporated into the robot arms as a possible means of tracking their motion.
In accordance with the preferred embodiment of the invention, a robot, with wheels, that contains onboard electronics and sensing equipment which make it possible for the robot to accurately decide where to move attached robotic arms and arm attachments either singularly or as a plurality, which robot can be guided by an operator over rows of crops. The robot contains computer and artificial intelligence hardware and software electronics and algorithms that make decisions customized to each plant or area that is operated on by the robot. The robot uses one or more means of sensing its environment, such as machine vision, ultrawideband radar, ultrasonics, parallax based laser rastering, touch and/or chemical sensing. Depending on the application, it may require more than one sensing technology operating together for acceptable field performance. For example, the robot may require machine vision, radar, and laser raster devices that utilize parallax on the reflection of the laser beam, with all three types of data integrated together by the electronics, to operate reliably in the field. Additionally, it may be useful for the robot to utilize touch sensors to confirm its map of the environment it operates in. Chemical sensors are also useful in some situations to locate and confirm the identity of an agricultural item.